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The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even If timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)n This action is FINAL. 2b)^ This action is non-final. 

3) n Since this application is in condition for allowance except for formal nnatters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-41 is/are pending in the application. 

4a) Of the above claim{s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-41 is/are rejected. 

Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 0 The specification is objected to by the Examiner. 

10)n The drawing(s) filed on is/are: a)n accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
11 )□ The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) n Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-{d) or (f). 

a)nAII b)n Some*c)n None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 1 9(e) (to a provisional application). 

a) n The translation of the foreign language provisional application has been received. 

15) n Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1 ) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary {PTO-41 3) Paper No(s). . 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Infomial Patent Application (PTO-1 52) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) Q Other: 
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DETAILED ACTION 



1 . Claims 1 -41 are presented for examination. 

Claim Objections 

2. Claim 24 is objected to because of the following informalities: Two duplicate 
words "with the" on line 8. Appropriate correction is required. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 



3. Claims 1-3,11, 27-28 and 31 are rejected under 35 U.S.C. 1 02(b) as being 

anticipated by Dimitrova et al. (US. Patent No. 5,870,754). 

Regarding on claims 1,11 and 27, Dimitrova teaches a method comprising: 
Initiating a search for images based (video clip) on a at least one query keyword 

in an query (news) (col. 22, lines 10-24); and 

Identifying, during the search, first images (56) (fig. 12) having associated 

keywords that match the query keyword and second images (52) (fig. 12) that contain 

low-level features similar to those of the first images (col. 9, lines 44-67). 



states. 
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Regarding on claims 2 and 28, Dimitrova teaches ranking the first and second 
images (col. 22, lines 20-24). 

Regarding on claims 3 and 31 , Dimitrova teaches presenting the first (56) and 
second images (52) (fig. 12). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

4. Claims 4-10, 12-23 and 33-41 are rejected under 35 U.S.C. 103(a) as being 

unpatentable over Dimitrova et al. (US. Patent No. 5,870,754) in view of Graham et al. 

(US. Patent No. 6,369,811). 

Regarding on claim 4, Dimitova presenting the first and second images to a user 

(col. 22, lines 10-36); however, Dimitova does not explicitly teach monitoring feedback 

from the user as to which of the first and second images are relevant to the query. On 

the other hand, Graham teaches, "the modification may occur in the background or may 

involve explicit user feedback input. The locations of concept if interest determined by 

pattern identification stage 620 are monitor by profile updating stage 634. Profile stage 

624 notes the proximity of other keywords and key phrase within each analyzed 
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document to the location of concepts of interest, the structure and contents of belief 
system 700 are updated in the back ground without the user input by profile updating 
data" (col. 7, lines 52-65). This teaches monitoring the feedback from the user or by the 
system. Therefore, it would have been obvious to one ordinary skill in the art at the time 
of the invention was made to modify the teaching of Graham into Dimitova in order to 
provide relevant feedback from the user based on the user assessment of the retrieval 
images. 

Regarding on claims 5 and 32, Dimitrova teaches presenting the first and second 
images to a user (coL 22, lines 10-36); However, Dimitova does not explicitly teach 
receiving feedback from the user as to whether the first an second images are relevant 
to the query; and learning how the first and second images are identified based on the 
feedback from the user. On the other hand, Graham teaches, " The modification may 
occur automatically in the back ground or may involve explicit user feedback input" (col. 
7, lines 52-54). This teaches the modification to the keyword to the concept or images. 
In addition, Graham teaches, "profile updating stage 624 notes the proximity of other 
keywords and key phrase within each analyzer document to the locations of concepts of 
interest. If particular keyword and key phrase are always near a concept of interest, the 
structure and content of belief system 700 are updated in the background without user 
input by profile updating stage 624" (col. 7, lines 56-62). This teaches the system learn 
how to assign new keyword to the concept or image. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention was made to employ 
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the learning technique of Graham into Dimitrova because by allowing the system 
intelligently learning from the user feedback to introduce the new connection between 
keyword and image to allow the user efficiently to retrieve better search results. 

Regarding on claims 6 and 33, Dimitrova teaches presenting the first and second 
images to a user (col. 22, lines 10-36); however, Dimitrova does not explicitly teach 
receiving the user feedback for the user as to which of the first and second images are 
relevant to the query; and refining the search to identify additional images that contain 
low-level features similar to those of the images indicated by the user as being relevant 
to the query. On the other hand, Graham teaches, "the modification may occur 
automatically in the background or may involve explicit user feedback input" (col. 7, 
lines 52-54). In addition, Graham also teaches, "the affect is to automatically refine the 
patterns searched" (col. 7, lines 3-4). This teaches the system receiving user feedback 
input and refines the search. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to modify the teaching of Graham 
into Dimitrova in order to provide relevant feedback from the user and also allow the 
user to refine the search when the retrieval results are not relevant. 
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Regarding on claims 7 and 34, Dimitrova teaches presenting the first and second 
images to a user (col. 22, lines 10-36); however, Dimitrova does not explicitly teach 
receiving feedback from the user as to which of the first and second images are relevant 
to the query; and assigning a large weight to an association between the query keyword 
and the images deemed relevant by the user. On the other hand, Graham teaches, "the 
modification may occur automatically in the background or may involve explicit user 
feedback input" (col. 7, lines 52-54). In addition, Graham also teaches, "changing 
probability values, introducing a new connection between a subconcept and concept, or 
introducing a new keyword or key phrase" (col. 7, lines 62-65). This teaches using the 
user's feedback to change or assign the new weight for the keyword or images to create 
the new connection between the keyword and the image or the document. Therefore, it 
would have been obvious to one ordinary skill in the art at the time of the invention was 
made to modify the teaching of Graham into Dimitrova in order to allow the system to 
adjust the weighting process to determine which of the images are more relevant to the 
search query to enable the user to retrieve the better image for the next search. 

Regarding on claims 8 and 35, Dimitrova teaches grouping the low-level features 
of the images deemed by relevant by the user (54, fig. 12). 
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Regarding on claims 9 and 36, Dimitrova teaclies presenting the first and second 
images to a user (col. 22, lines 10-36); However, Dimitrova does not explicitly teach 
receiving feedback from the user identifying an example image as less or irrelevant to 
the query for refinement of the search; and assign a small weight to an association 
between the query keyword and the example image. Graham teaches, "the modification 
may occur automatically in the background or may involve explicitly user feedback 
input" (col. 7, lines 52-54). This teaches the system is receiving user' feedback. In 
addition, Graham also teaches, "user 504 indicates the selected keywords or key 
phrases to be a little interest, the probability values connecting these keywords or key 
phases to the concept decrease" (col. 8, lines 10-13). This teaches the value of the 
keyword associated with the concept is decreased if the keyword is not descriptive. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to modify the teaching of Dimitrova into Graham because by 
deceasing the value of a keyword associated with concept or image would allow the 
user to search again with the same keyword without receiving the same irrelevant 
concept or image. 

Regarding on claims 10 and 37, Dimitrova teaches identifying additional images 
with low-level features similar to those of the example image (54. fig. 12). 
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Regarding on claims 12, 15, 39 and 41, Dimitrova teaches a method comprising: 

Permitting entry of both keyword-based queries (query by keyword) and content- 
based queries (query by image) (fig. 12); 

Finding images using both semantic-based image retrieval and low-level feature- 
based image retrieval (col. 22, lines 10-24); 

Presenting the images to a user so that the user can indicate whether the images 
are relevant (col. 22, lines 19-20); and 

Dimitrova does not explicitly teach conducting semantic-based relevance 
feedback and low-level feature-based relevance feedback in an integrated fashion. 
However, Graham teaches, "The first window 902 may not be necessary when adjusting 
the relevance level for a keyword or key phrase that is already a part of belief system. 
After the user selects a concept in the first feedback window 902, a second window 
feedback window 904 is displayed for selecting the degree of relevancy" (col. 9, lines 
1 0-31 ). This teaches the user or the system can modify the association for the keyword 
to a concept or images. Therefore, it would have been obvious to one ordinary skill in 
the art at the time of the invention was made to modify the teaching of Graham into 
Dimitrova because by utilizing the relevancy feedback would allow user to change the 
association between the keyword and the concept or image to allow the user to retrieve 
better result. 
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Regarding on claims 13 and 40, Dimitrova teaches ranking (ordering) the images 
(col. 22, lines 20-24). 

Regarding on claim 14, Dimitrova teaches using images indicated (56, fig. 12) as 
being relevant to find additional images (54, fig. 12). 

Regarding on claims 16 and 19, Dimitrova teaches a method comprising: 

associating keywords with images to form keyword-image links (keyword to 
retrieve image) (query by keyword, fig. 12); 

Assigning weight to the keyword-image links (col. 16, lines 21-23); 

Presenting a result set of images obtained from an image retrieval search based 
on a query (col. 22, lines 5-24); 

Dimitrova does not explicitly teaches receiving feedback from the user as to 
whether the images in the result set are relevant to the query; and modifying the weight 
according to the user feedback. However, Graham teaches, "The first window 902 may 
not be necessary when adjusting the relevance level for a keyword or key phrase that is 
already a part of belief system. After the user selects a concept in the first feedback 
window 902, a second window feedback window 904 is displayed for selecting the 
degree of relevancy" (col. 9, lines 10-31 ). This teaches the user or the system can 
modify the association for the keyword to a concept or image. In addition, Graham 
teaches, "if the particular keywords and key phrases are always near a concept of 
interest, the structure and contents of belief system 700 are updated stage 24. This 
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could means changing probability values, introducing a new connection between a 
subconcept and concept, or introducing a new keyword or key phrase. This teaches the 
keywords or key phrases are assigned with the new higher probability when the concept 
or image is relevance to the keyword. Therefore, it would have been obvious to one 
ordinary skill in the art at the time of the invention was made to modify the teaching of 
Graham into Dimitrova because by utilizing the relevancy feedback would allow user to 
change the association between the keyword and the concept or image by adjusting the 
weight. 

Regarding on claim 17, Graham teaches the modifying comprises increasing the 
weight of a keyword-image link for images deemed by the user as more relevant to the 
query (col. 7, lines 62-65). 

Regarding on claim 18, Graham teaches the modifying comprises decreasing the 
weight of a keyword-image link for images deemed by the user as less relevant to the 
query (col. 8, lines 10-13). 
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Regarding on claims 20 and 23, Dimitrova teaches the a method comprising: 
Presenting a result set of images that are returned that are returned from an 
image retrieval search of query having at least one keyword (col. 22, lines 10-24); 

Dimitrova does not explicitly teach monitoring feedback from a user as to 
whether the image in the result set are relevant to the query; in an event that the user 
selects at least one image as being relevant to the query, associating the keyword in the 
query with the selected image to form a first keyword-image association and assigning a 
comparatively large weight to the first key-image association; and in an event that the 
user identifies an example image for refinement of the search, associating the keyword 
in the query with the example image to form a second keyword-image association and 
assigning a comparatively small weight to the second keyword-image association. 
However, Graham teaches, "The first window 902 may not be necessary when adjusting 
the relevance level for a keyword or key phrase that is already a part of belief system. 
After the user selects a concept in the first feedback window 902, a second window 
feedback window 904 is displayed for selecting the degree of relevancy" (col. 9, lines 
1 0-31 ). This teaches the user or the system can modify the association for the keyword 
to a concept or image. In addition, Graham teaches, "if the particular keywords and key 
phrases are always near a concept of interest, the structure and contents of belief 
system 700 are updated stage 24. This could means changing probability values, 
introducing a new connection between a subconcept and concept, or introducing a new 
keyword or key phrase. This teaches the keywords or key phrases are assigned with 
the new higher probability when the concept or image is relevance to the keyword. 
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Further more, Graham also teaches, "if, on the other hand, user 504 indicates the 
selected keywords or key phrases to be of little interest, the probability values 
connecting these keyword or key phrase to the concept decrease" (col. 8, lines 10-14). 
This teaches the probability value is decreased when the keywords or key phrases are 
less relevance to the concept or image. Therefore, it would have been obvious to one 
ordinary skill in the art at the time of the invention was made to modify the teaching of 
Graham into Dimitrova because utilizing the relevance feedback would change the 
probability values of the association between the keyword or key phrase and concept or 
image to enable the user to obtain the better search using the same keyword. 

Regarding on claim 21, Dimitrova teaches conducting both content-based image 
retrieval and semantic-based image retrieval (col. 22, lines 10-24 and fig. 12) 

Regarding on claim 22, Graham teaches presenting the result set of images 
(concepts) in a user interface, the user interface (user feedback input) facilitating the 
user feedback by allowing the user to indicate which images (concepts) are more 
relevance (col. 7, lines 51-65) and which images are less relevant (col. 8, lines 10-13). 



Application/Control Number: 09/702,292 Page 13 

Art Unit: 2172 

Regarding on claim 38, Dimitrova teaclies structure stored on one or more 
computer-readable media comprising: 

Multiple images files (video clips or images) (col. 22, line 9); 

Multiple l^eywords (using keywords search for images) (query keyword search, 

fig. 12) 

A semantic network to associated the keywords with the image files, the 
semantic network defining individual keyword-image links that associate a particular 
keyword with a particular image file (col. 22, lines 10-24), each keyword-image link 
having a weight indicative of how relevant the particular keyword is to the particular 
image file. Dimitrova does not explicitly teach each key word having the weight indicate 
how relevant the particular keyword it to the particular image file. However, Graham 
teaches, "this could mean changing probability values, introducing a new connection 
between a subconcept and concept, or introducing a new keyword or key phrase" (col. 
7, lines 62-65). This teaches there is a weight existed between the keywords or key 
phases and the concept or image before using the user relevance feedback to change 
the connection between keywords or key phrase with the concept or image. Therefore, 
it would have been obvious to one ordinary skill in the art at the time of the invention 
was made to modify the teaching of Dimitrova into Graham because by assigning the 
weight to the keyword would allow the system to determine the irrelevant or relevance 
when the value of the keyword increase or decrease based on the user relevance 
feedback. 
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5. Claims 29 and 30 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Dimitrova et al. (US. Patent No. 5,870,754) in view of Stuckey et al. (US. Patent 
No. 5,721,938). 

Regarding on claim 29, Dimitrova does not explicitly teaches the query handler 
comprises a natural language parser; however, Stuckey teaches, "present program 
provides a natural text parser, which may be used for all natural languages" (col. 3, lines 
22-24). This teaches the query is inputted by the natural language. Therefore, it would 
have been obvious to one ordinary skill in the art at the time of the invention was made 
to modify the teaching of Stuckey into Dimitrova because utilizing the natural language 
parser would allow the user to input the natural language to search and to provide a 
user-friendly system. 

Regarding on claim 30, Stuckey teaches the query handler comprises: 

A parser to parse text-based queries (col. 3, lines 22-24); and 

A concept hierarchy to define various categories of images (col. 12, lines 22-25). 
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6. Claims 24-26 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Abdel-Mottaleb et al. (US. Patent No. 6,285,995) in view of Kopec et al. (US. Patent 
No. 5.594,809). 

Regarding on claim 24, Abdel-Mottaleb teaches a method comprising: 

Computing, for each category, a representative feature vectors of a set of 
existing images with the category (col. 5, lines 15-20); 

Determining a set of representative keywords (keywords adds to image) that are 
associated with the existing images In each category (col. 5, lines 10-12); 

Comparing (comparison unit 128), for each new image, the low-level feature 
vectors of the new image to the representative feature vectors of the existing images in 
each category to identify a closet matching category (col. 5, lines 45-65); and 

Abdel-Mottaleb does not explicitly teach labeling the new image with the set of 
representative keywords associated the closet matching category; however, Kopec 
teaches, "in input keyword image sample into image segments, and providing labels for 
the image segments" (col. 7, lines 63-65). This teaches the image segment is labeled 
by the keyword. Therefore, it would have been obvious to one ordinary skill in the art at 
the time of the invention was made to employ the teaching of Kopec into Abdel-Mottaleb 
in order to match the image with the keyword to allow the user search for these images 
with the same keyword. 

Regarding on claim 25, Abdel-Mottaleb teaches using feedback to selectively 
add and/or remove keywords from the new image (col. 5, lines 10-12). 
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Regarding on claim 26, Abdel-Mottaleb teaches placing the labeled new images 
into a holding category (col. 5, lines 15-20). 

Abdel-Mottaleb does not explicitly teach evaluating the labeled new images in the 
holding category to determine if any of the keywords associated with the labeled new 
image match the representative keywords from each category; and assigning the 
labeled new image to the category that best matches the keywords associated with the 
labeled with the labeled new image. 

Spec teaches evaluating the labeled new images in the holding category to 
determine if any of the keywords associated with the labeled new image match the 
representative keywords from each category (col. 7, lines 63-65); and assigning (input 
keyword image) the labeled new image to the category that best matches the keywords 
associated with the labeled with the labeled new image (col. 7, lines 64-65). 

Therefore, it would have been obvious to one ordinary skill in the art at the time 
of the invention was made to employ the teaching of Kopec into Abdel-Mottaleb in order 
to match the image with the keyword to allow the user search for these images with the 
same keyword. 
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Conclusion 



7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Baoquoc N. To whose telephone number is (703) 305- 
1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM - 4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kim Y. Vu can be reached at (703) 305-4393. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 
Commissioner of Patents and Trademarks 
Washington, D.C. 20231. 

The fax numbers for the organization where this application or proceeding Is 
assigned are as follow: 

• (703) 746-7238 [After Final Communication}] 

• (703) 746-7239 [Official Communication] 

• (703) 746-7240 [Non-Official Communication] 
Hand-delivered responses should be brought to: ^ / 



Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 
Fourth Floor (Receptionist). 



JEAN M. CORRIELUS 
PRIMARY EXAMINER 




Baoquoc N. To 
November 28, 2002 



